In recent years, a full HD (1920 pixels×1080 pixels) television is becoming general, besides, a display panel having a high resolution such as 4k2k (4096 pixels×2048 pixels), 8k4k (8192 pixels×4096 pixels) and the like is also developed. On the other hand, SD (720 pixels×480 pixels under NTSC, 720 pixels×576 pixels under PAL) contents are present in large quantities. Accordingly, an image enlargement technology for displaying low resolution contents on a high resolution display device is necessary.
Conventionally, various image enlargement methods are proposed, in which there is a method that uses wavelet transform.
Here, an overview of the wavelet transform is described. FIG. 17 is a view that schematically shows an original image and a wavelet transform image. When discrete wavelet transform (DWT) is applied to an original image IMG, a wavelet transform image is obtained, which is composed of sub-band components IMG_LL, IMG_HL, IMG_LH, and IMG_HH. The image size of each sub-band component is ¼ of the original IMG.
A low pass filter process and a sampling process for obtaining the ½ size are applied to the original image IMG in a horizontal direction, then, to whose result the low pass filter process and the sampling process for obtaining the ½ size are applied in a vertical direction, whose result is the sub-band component IMG_LL. Here, the sub-band component IMG_LL may be a result that is obtained by applying the low pass filter process and the sampling process for obtaining the ½ size to the original image IMG in the vertical direction, then, applying to whose result the low pass filter process and the sampling process for obtaining the ½ size in the horizontal direction.
A high pass filter process and the sampling process for obtaining the ½ size are applied to the original image IMG in the horizontal direction, then, to whose result the low pass filter process and the sampling process for obtaining the ½ size are applied in the vertical direction, whose result is the sub-band component IMG_HL. Accordingly, the sub-band component IMG_HL represents an image in which a high-frequency component in the horizontal direction of the original image IMG is extracted, that is, an image that reflects edge information that faces in the vertical direction. Here, the sub-band component IMG_HL may be a result that is obtained by applying the low pass filter process and the sampling process for obtaining the ½ size to the original image IMG in the vertical direction, then, applying to whose result the high pass filter process and the sampling process for obtaining the ½ size in the horizontal direction.
The low pass filter process and the sampling process for obtaining the ½ size are applied to the original image IMG in the horizontal direction, then, to whose result the high pass filter process and the sampling process for obtaining the ½ size are applied in the vertical direction, whose result is the sub-band component IMG_LH. Accordingly, the sub-band component IMG_LH represents an image in which a high-frequency component in the vertical direction of the original image IMG is extracted, that is, an image that reflects edge information that faces in the horizontal direction. Here, the sub-band component IMG_LH may be a result that is obtained by applying the high pass filter process and the sampling process for obtaining the ½ size to the original image IMG in the vertical direction, then, applying to whose result the low pass filter process and the sampling process for obtaining the ½ size in the horizontal direction.
The high pass filter process and the sampling process for obtaining the ½ size are applied to the original image IMG in the horizontal direction, then, to whose result the high pass filter process and the sampling process for obtaining the ½ size are applied in the vertical direction, whose result is the sub-band component IMG_HH. Accordingly, the sub-band component IMG_HH represents an image (an image in which a high frequency component in an oblique direction of the original image IMG is extracted) in which a high-frequency component in the horizontal direction and a high frequency component in the vertical direction of the original image IMG are extracted, that is, an image that reflects edge information that faces in the oblique direction. Here, the sub-band component IMG_HH may be a result that is obtained by applying the high pass filter process and the sampling process for obtaining the ½ size to the original image IMG in the vertical direction, then, applying to whose result the high pass filter process and the sampling process for obtaining the ½ size in the horizontal direction.
By applying inverse discrete wavelet transform (IDWT) to the wavelet transform image that is composed of the sub-band components IMG_LL, IMG_HL, IMG_LH, and IMG_HH, the original image IMG is obtained.
Next, an overview of an image enlargement method using the wavelet transform is described. FIG. 18 is a view that schematically shows the overview of the image enlargement method that uses the wavelet transform.
An input image IMG_IN which is an enlargement target image is regarded as the sub-band component IMG_LL of a wavelet transform image. There is no information about the remaining three sub-band components (sub-band components in which a high frequency component is contained) IMG_HL, IMG_LH, and IMG_HH, accordingly, sub-band components IMG_HL (0), IMG_LH (0) and IMG_HH (0), all pixel values of which are 0, are used. According to this, the input image IMG_IN is regardable as a wavelet transform image that is composed of the sub-band components IMG_IN, IMG_HL (0), IMG_LH (0) and IMG_HH (0).
By applying the inverse discrete wavelet transform to the wavelet transform image that is composed of the sub-band components IMG_IN, IMG_HL (0), IMG_LH (0) and IMG_HH (0), an output image IMG_OUT is obtained, which is an enlarged image and has the number of pixels that is 4 times the input image IMG_IN. However, the wavelet transform image composed of the sub-band components IMG_IN, IMG_HL (0), IMG_LH (0) and IMG_HH (0) does not have information about a high frequency component, accordingly, the output image IMG_OUT which is the enlarged image is prone to become blurred.
To the contrary, if there is suitable information about the three sub-band components (sub-band components in which the high frequency component is contained) IMG_HL, IMG_LH, and IMG_HH, it is possible to solve the problem that the enlarged image is prone to become blurred. Besides, by performing gain adjustment of the sub-band component IMG_HL and thereafter performing the inverse discrete wavelet transform, it is possible to obtain an enlarged image in which an edge facing in the vertical direction is accentuated; by performing gain adjustment of the sub-band component IMG_LH and thereafter performing the inverse discrete wavelet transform, it is possible to obtain an enlarged image in which an edge facing in the horizontal direction is accentuated; and by performing gain adjustment of the sub-band component IMG_HH and thereafter performing the inverse discrete wavelet transform, it is possible to obtain an enlarged image in which an edge facing in the oblique horizontal direction is accentuated.
Methods for obtaining the information about the three sub-band components (sub-band components in which the high frequency component is contained) IMG_HL, IMG_LH, and IMG_HH are proposed by a patent literature 1, a patent literature 2, and non-patent literatures 1-3.
The image enlargement method proposed by the patent literature 1 is described with reference to FIG. 19. First, by applying the discrete wavelet transform to the input image IMG_IN that is an enlargement target, a wavelet transform image is obtained, which is composed of sub-band components LL2, HL2, LH2, and HH2. Then, by using information about the sub-band components LL2, HL2, LH2, and HH2, the three sub-band components (sub-band components in which the high frequency component is contained) IMG_HL, IMG_HH, and IMG_HH are predicted. Here, a coefficient used for the prediction is obtained from learning. And, the input image IMG_IN, which is the enlargement target, is regarded as the sub-band component IMG_LL of the wavelet transform image; a combination of the sub-band component IMG_LL and the predicted three sub-band components (sub-band components in which the high frequency component is contained) IMG_HL, IMG_LH, and IMG_HH is regarded as the wavelet transform image, and the inverse discrete wavelet transform is performed, whereby the output image IMG_OUT which is an enlarged image is obtained. Here, also the image enlargement method proposed by the patent literature 3, like the image enlargement method proposed by the patent literature 1, predicts the three sub-band components (sub-band components in which the high frequency component is contained) IMG_HL, IMG_LH, and IMG_HH.
Next, the image enlargement method proposed by the patent literature 2 is described with reference to FIG. 20. An enlargement target image is regarded as the sub-band component IMG_LL of a wavelet transform image. Then, by using a filter shown in FIG. 20 (a), an edge component in a vertical direction of the enlargement target image is extracted and is regarded as the sub-band component IMG_HL that is deficient; by using a filter shown in FIG. 20 (b), an edge component in a horizontal direction of the enlargement target image is extracted and regarded as the sub-band component IMG_LH that is deficient; by using a filter shown in FIG. 20 (c), an edge component in an oblique direction of the enlargement target image is extracted and regarded as the sub-band component IMG_HH that is deficient; a combination of the above four is regarded as the wavelet transform image, and the inverse discrete wavelet transform is performed, whereby an enlarged image is obtained.
Next, the image enlargement method proposed by the non-patent literature 2 is described. The non-patent literature 1 describes a method for predicting the three sub-band components (sub-band components in which a high frequency component is contained) by assuming Sparsity constraint, repeatedly performing calculations until convergence and predicting an edge; the non-patent literature 2 applies this method to the image enlargement. The non-patent literature 3 improves the method described in the non-patent literature 1, reduces the number of convergences to 2 and performs the image enlargement.
According to the image enlargement method proposed by the patent literature 1, there is no guarantee of correctly predicting a sub-band component which contains a high frequency component of an image that is not taught. Besides, according to the image enlargement method proposed by the patent literature 1, it is impossible to perform the image enlargement and sufficient edge accentuation at the same time. Because of this, there is a risk that the enlarged image is blurred and becomes unclear. Here, the image enlargement method proposed by the patent literature 3 has the same problem as the image enlargement method proposed by the patent literature 1, further has also a problem that a large quantity of calculations are required.
According to the image enlargement method proposed by the patent literature 2, determining from an exemplified edge detection filter (see FIG. 20), it is impossible to obtain suitable information about a sub-band component in which a high frequency component is contained. Even in a case where the exemplified edge detection filter is used as it is to perform edge accentuation, an overshoot and Jaggy occur, further, when the inverse discrete wavelet transform is performed, the image size is enlarged by two times in both of the horizontal direction and the vertical direction, accordingly, the Jaggy spreads. Accordingly, in the enlarged image obtained by the image enlargement method proposed by the patent literature 2, unnatural Jaggy occurs in the accentuated edge.
The image enlargement methods proposed by the non-patent literature 2 and the non-patent literature 3 have a problem that it is impossible to accentuate a weak edge which is equal to or smaller than a threshold value under a sparsity constraint condition. Besides, according to the image enlargement methods proposed by the non-patent literature 2 and the non-patent literature 3, in the prediction of edge information, it is necessary to repeatedly perform a plurality of calculations until convergence, accordingly, there is also a problem that the amount of calculations is large and a delay occurs.
An image enlargement device which is able to solve the above problems is invented by the inventor of the present application and is already filed by the inventor as a patent application (Japanese patent application No. 2009-225995). FIG. 21 shows an example of the image enlargement device proposed by the Japanese patent application No. 2009-225995.
The image enlargement device shown in FIG. 21 includes: a Lanczos3 filter 101; a discrete wavelet transform portion 102; multipliers 103 to 105; an inverse discrete wavelet transform portion 106; and a control portion (not shown) that incorporates a rewritable non-volatile memory.
The control portion reads, from the non-volatile memory, constant settings for the Lanczos3 filter 101, the discrete wavelet transform portion 102, and the inverse discrete wavelet transform portion 106, and sets constants into the Lanczos3 filter 101, the discrete wavelet transform portion 102, and the inverse discrete wavelet transform portion 106.
The Lanczos3 filter 101 generates an enlarged image IMG_UP that is obtained by enlarging the input image IMG_IN in both of the horizontal direction and the vertical direction.
The discreet wavelet transform portion 102 applies the discrete wavelet transform to the enlarged image IMG_UP, thereby generating the sub-band components IMG_LL, IMG_HL, IMG_LH, and IMG_HH.
Besides, the control portion reads, from the non-volatile memory, a gain value G_HL that corresponds to accentuation strength of an edge which faces in the vertical direction; a gain value G_LH that corresponds to accentuation strength of an edge which faces in the horizontal direction; and a gain value G_HH that corresponds to accentuation strength of an edge which faces in the oblique direction; supplies the gain value G_HL to the multiplier 103; the gain value G_LH to the multiplier 104; and the gain value G_HH to the multiplier 105. The multiplier 103 supplies a product, which is obtained by multiplying the sub-band component IMG_HL and the gain value G_HL, to the inverse discrete wavelet transform portion 106. The multiplier 104 supplies a product, which is obtained by multiplying the sub-band component IMG_LH and the gain value G_LH, to the inverse discrete wavelet transform portion 106. The multiplier 105 supplies a product, which is obtained by multiplying the sub-band component IMG_HH and the gain value G_HH, to the inverse discrete wavelet transform portion 106. Here, the sub-band component IMG_LL is supplied as it is to the discrete wavelet transform portion 106.
The inverse discrete wavelet transform portion 106 regards the sub-band component IMG_LL as the sub-band component IMG_LL of an wavelet transform image and regards a combination of the sub-band component IMG_LL and the three sub-band components IMG_HL•G_HL, IMG_LH•G_LH, and IMG_HH•G_HH after the gain process as a wavelet transform image, and performs the inverse discrete wavelet transform, thereby generating an enlarged image IMG_SYNTH. The enlarged image IMG_SYNTH becomes an output from the image enlargement device shown in FIG. 21.
During the image enlargement process by the Lanczos3 filter 101, it is possible to boost a weak high frequency component (somewhat attenuated high frequency component), however, it is impossible to adjust the strength of the weak high frequency component. Nevertheless, according to the image enlargement device shown in FIG. 21, it is possible to adjust the edge accentuation strength by applying the gain process to the sub-band components IMG_HL, IMG_HH, and IMG_HH, accordingly, it is possible to obtain an image that has less Jaggy and a high quality.    PLT1: JP-A-2000-215305 (paragraph [0035], FIG. 5)    PLT2: JP-A-2001-8027 (paragraph [0041], FIG. 1, FIG. 6)    PLT3: JP-A-1995-152907 (abstract)    NPLT1: G. Guleryuz, “Predicting wavelet coefficients over Edges using estimates based on nonlinear approximants,” Proc. data compression conference, IEEE DCC-04, 2004.    NPLT2: C. S. Boon, O. G. Guleryuz, T. Kawahara, and Y. Suzuki, “Sparse super-resolution reconstructions of video from mobile devices in digital TV broadcast applications,” Proc. SPIE conference on applications on of digital image processing XXIX, San Diego, 2006.    NPLT3: S. Kanumuri, O. G. Guleryuz and M. R. Givanlar. “Fast super-resolution reconstructions of mobile video using warped transforms and adaptive thresholding,” Proc. SPIE conference on applications of digital image processing XXX, 2007.